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Abstract: We study convolution algebras associated with Heckman- 
Opdam polynomials. For root systems of type BC we derive three con- 
tinuous classes of positive convolution algebras (hypergroups) by interpo- 
lating the double coset convolution structures of compact Grassmannians 
U/K with fixed rank over the real, complex or quaternionic numbers. 
These convolution algebras are linked to explicit positive product formu- 
las for Heckman-Opdam polynomials of type BC, which occur for certain 
discrete multiplicities as the spherical functions of U /K. These results 
complement those of a recent paper by the second author for the non- 
compact case. 
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1 Introduction 

In the theory of multivariable hypergeometric functions and polynomials of 
Heckman, Cherednik and Opdam, the existence of product formulas and posi- 
tive convolution algebras is in general unsolved. In |18j , three continuous series 
of positive convolution algebras having Heckman-Opdam hypergeometric func- 
tions as multiplicative functions were obtained by interpolating geometric cases 
in an explicit way, namely the product formulas for the spherical functions of 
non-compact Grassmannians. In these full picture of harmonic analysis 

for the hypergeometric transform could thus be obtained. The present paper 
extends these results to the dual situation related to compact Grassmannians 
and convolution algebras for three continuous series of Heckman-Opdam Jacobi 
polynomials of type BC. 
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To be specific, we consider the compact Grassmann manifolds U/K where 
U = SO(p + q), SU(p + q) or Sp(p + q) and K = SO(p) x SO(q), S(U(p) X U(q)) 
or Sp(p) x Sp(q), respectively. These are dual to the noncompact Grassman- 
nians studied in [IS]. Following the procedure in loc.cit., we write down the 
product formula for their spherical functions in a way which allows analytic 
continuation with respect to the dimension parameter p, the rank q being fixed. 
The spherical functions are Heckman-Opdam Jacobi polynomials of type BC 
with certain discrete multiplicities, and our continuation gives an explicit prod- 
uct formula for an interpolated continuous range of multiplicities. This formula 
in part generalizes Koornwinder's product formula for Jacobi polynomials [12j 
to higher rank. Naturally, it is similar to the non-compact case, but direct an- 
alytic continuation from the non-compact to the compact case seems to be not 
feasible. 

The compact case is easier in some respect (identifying the dual space for 
instance), but also needs some special care when studying the geometric back- 
ground. We obtain commutative hypergroup algebras on the fundamental alcove 
of the associated affine reflection group, with the associated Heckman-Opdam 
Jacobi polynomials as characters. 

The organisation of this paper is as follows: In Section [5] we recall some 
basics of trigonometric Dunkl theory. Section [3] is a short summary of the 
necessary background from the theory of symmetric spaces. After this we start 
in Section 4 with the compact Grassmannians U /K, identify their spherical 
functions with Jacobi polynomials, and use a KAK-type decomposition to make 
their product formula explicit. Following the idea of [18], this product formula 
is then analytically continued. Section 5 contains a review of the rank one case, 
and in Section 6, the related hypergroup structures on the fundamental alcove 
are studied. 

2 Fundamentals of Trigonometric Dunkl Theory 

This section is a short review of the fundamentals of trigonometric Dunkl the- 
ory which will be needed in this article. For details, we refer to the work of 
Heckman and Opdam ([6], [15], [16]). 

Let a be a g-dimensional Euclidean space with inner product (•, •), which is 
extended to a complex bilinear form on the complexification ac of a. We identify 
a with its dual space a* = Hom(o, K) via the given inner product. Let £ C a be 
a (not necessarily reduced) root system. For a £ £ we write a v := 2a/ (a, a) 
for the coroot of a and denote by s a (x) = x — (a v ,x)a the reflection in the 
hyperplane H a perpendicular to a. 

The reflections {s a : a € £} generate the Weyl group W = W{Ti). We 
define the root lattice Q := Z.S and the coroot lattice Q v = Z.E V . Further, 
we fix some positive subsystem S + of E, as well as a basis {a±, . . . a q } c S + 
of simple roots. An element A S a is called (strictly) dominant, if (A, cti) > 
(respectively > 0) for all i — l,...,q. We write a + := {A e a : (A, a v ) > 
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Va G S + } for the Weyl chamber of strictly dominant elements. 
For a £ £ and A € ac let 

x _ (A, a) 



(a, a) ' 

The weight lattice is given by 

A := {A £ a : X a £ Z for all a G £} 

and the set 

A+ := {A £ a : X a £ Z + for all a £ £+} 

is called the lattice of dominant weights. Here we use the notation Z + := 
{0, 1,2,.. .}. The positive root lattice Q + = Z + .£ + defines a partial ordering 
^ on a: 

\i r< A A - /i e Q + . 

This ordering is called the dominance ordering. 

2.1 Lemma. Let 7 G a+ &e dominant. Then ^ 7 for all w £ W . 

Proof. Lemma 10. 3B in [10]. □ 

2.2 Lemma. Let X, fi £ A + be dominant weights with fi ^ A. T/ien < |A|. 

Proof. Let A, /i G A + with /i X A. Then A + /i is also dominant and A — /1 is a 
sum of positive roots. Therefore 

0< (A + /i,A-/i) = |A| 2 -|/i| 2 . 

□ 

A multiplicity function is a VT-invariant map m : £ — * C, a 1— > m Q . We 
denote the set of multiplicity functions by M. . In this article we only consider 
non-negative multiplicities, i.e. m a > for all a £ S. Define 

p = p(m) := - m Q a. (2.1) 

QGE+ 

2.3 Definition. Let (£0 and m £ M. The Dunkl- Cherednik operator associ- 
ated with £ and m is given by 

T£=T(£,m) :=d$+ m a (a, g) - *_ 2a (1 - a a ) - (/?,£), 
where <9,e is the usual directional derivative and e A (£) := e^ A '^ for A,£ G Oc- 
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2-4 Remark. Heckman and Opdam use a slightly different notation. They con- 
sider a root system R with multiplicity k, which is connected to our notation 
via 

R = 2E, k 2a = -m a . 

Note that this implies further differences. Our notation comes from the theory 
of symmetric spaces. 

For fixed multiplicity m, the operators lg, £ S ac commute. Therefore the 
assignment £ i— > T(£, m) uniquely extends to a homomorphism on the symmetric 
algebra S(ac) over ac, which may be identified with the algebra of complex 
polynomials on ac- Let T(p,m) be the operator which corresponds in this way 
tope S(ac)- If p € S(ac) w , the subspace of VF-invariant polynomials on ac, 
then T(p, m) acts as a differential operator on the space of H^-invariant analytic 
functions on a. 

Fix a spectral parameter A G ac- Then according to fundamental results of 
Heckman and Opdam (see [6]), the so-called hypergeometric system 

T(p, m)tp = p(X)(p for all p e S{a c ) W 

has a unique VF-invariant solution tp = F\(m; •) = F(A,m; •) which is analytic 
on a and satisfies F\(m;0) = 1. Moreover, there is a M^-invariant tubular 
neighborhood U of a in ac such that F extends to a (single- valued) holomorphic 
function F : ac x Al rcg x U — * C. The function f (A, m; x) is VF-invariant in both 
A and x. It is called the hypergeometric function associated with E. For certain 
spectral parameters A, the functions F\ are actually trigonometric polynomials, 
the so-called Heckman-Opdam polynomials. In order to make this precise, we 
need some more notation. 

Let T := lin{e lA : A £ A} be the space of trigonometric polynomials as- 
sociated with A. Trigonometric polynomials are 7rQ v -periodic, and T(T C T. 
Consider the torus T — a/nQ y with the VF-invariant weight function 

w m := J] \e ia - e- ia \ ma . (2.2) 

Let 

M\ := J2 A e A+ 

new.x 

denote the W- invariant orbit sums. They form a basis of the space of W- 
invariant trigonometric polynomials T w . For A € A + the (Heckman-Opdam) 
Jacobi polynomials associated with S are now defined by 

.Pa = P\(m; ■) ■= J2 c ^M M m 
mga+, ^\ 

where the coefficients cx^im) are uniquely determined by the conditions 
(i) c\\(m) = 1 
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(ii) P\ is orthogonal to in L 2 (T;w m ) for all p £ A + with /x -< A. 
The Jacobi polynomials P\ form an orthogonal basis of I^TjWm) 1 *', the 
subspace of VF-invariant elements from L 2 (T, w m ). 

2. 5 Remark. Notice that our notation slightly differs from that of Heckman and 
Opdam (e.g. [BJ, [U]), namely by a factor i in the spectral variable. This choice 
of notation will be more convenient for our purposes. 

The connection between the Jacobi polynomials and the hypergeometric 
function is as follows: 

2.6 Lemma. (See '6]) For all x £ ac and A £ A + , 

F\ +p (m;ix) = c(A + p,m)P\(m;x), 
where the c-function c(A + p, m) = P\(m; 0) _1 is given by 



c(A + p,m) = Yl 



T(A Q + p a + \m a/2 )r(p a + \m a / 2 + \m a ) 



m ,, + r(A Q + p a + \m a/2 + \m a )T(p a + \m a / 2 ) 
We shall work with a renormalized version of the Jacobi polynomials, defined 

by 

R\{x) :— R\(m; x) := c(A + p, m)P\(m; x). (2-3) 

They satisfy R x (0) = 1. 

Let us identify a with M. q . Dividing the torus T = a/irQ^ by the action of the 
Weyl group W gives the fundamental alcove A$ = {x £ R 9 : 5 > x\ > x 2 > 
• • • > ^ 9 > 0}. We may consider the W- invariant trigonometric polynomials 
T w as functions on the fundamental alcove Ao . The Jacobi polynomials R\ are 
orthogonal with respect to the inner product 



(f,g)m = l_f{x)g{x)w m {x)dx, 

JAo 

but they are not orthonormal. We put 



'* = K- (2 - 4) 



3 Compact symmetric spaces and their spheri- 
cal functions 

In this section we recall some basics from the theory of symmetric spaces. Stan- 
dard references are the books of Helgason [7] and [5] . 

Let U be a compact semisimple and connected Lie group with Lie algebra 
u. Let ii" be a closed subgroup such that M := U/K is connected and there 
exists an involutive automorphism 8 : U — > U with 11® = K. Here U e = {u £ 
U : 6(u) = u} and U® denotes the identity component of U 6 . Then M = U/K 
is a compact symmetric space. 
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The derivation of 9 gives an involution of the Lie algebra u, which we also 
denote by 9. The corresponding Cartan decomposition is given by u = t © q. 
Let b C q be a maximal abelian subspace. Denote by G the analytic subgroup 
of the complexification Uc with Lie algebra g := 6 © iq. We have K C G. Put 
p := iq. Then a := ib is a maximal abelian subspace of p. Denote by 9c the 
analytic continuation of 9 to t/c and let r = Then r is a Cartan involution 
of G and K = G T . The symmetric space G/K is called the noncompact dual 
of U/K. Denote by £ := S(g, a) the restricted root system and by S + a fixed 
subset of positive restricted roots. 

Recall that for an arbitrary Lie group G with compact subgroup K, a spheri- 
cal function on G is a nonzero, fC-biinvariant function ip : G — > C which satisfies 
the product formula 

<p(g)ip(h) = / (p(gkh)dk for all g,h<EG, (3-1) 

where dfc denotes the normalized Haar measure on K. 

Assume now that (U, K) is as above, and U is also simply connected. Then 
the spherical functions on U are indexed by the set 

(vea : v a := j^eZ+ (VaeS+)l, (3.2) 
I {a- a) J 

which coincides with the set A + (U/K) of (restrictions of ) highest weights of 
-R"-spherical irreducible representations of U. Recall that an irreducible unitary 
representation <&:[/—► GL(V) is called K- spherical, if there exists a if-fixed 
vector. More precisely, the following holds ([5], Theorem 3.4, Ch. IV and 
Theorem 4.1, Ch. V): 

3.1 Proposition. Assume that U is simply connected. Let [i G A + (C//if) and 

GL(V) be a spherical representation of U . Choose a K-fixed vector 
€ V with Wv^W = 1. Then 

is a spherical function on U . Conversely, every spherical function on U is of 
this form for some unique fi € A + (U / K). 

If U is not simply connected, then the spherical functions on U are in general 
indexed by a subset of {v e a : i/ a eZ + (Va G 

We assume again that U is simply connected, and G/K is the concompact 
dual of U/K. Then there is a close connection between the spherical functions 
on G and those on U, which is given by the following important proposition. 

3.2 Proposition. Every spherical function (p\ on G (A G Oc) is an analytic 
function. It extends to a holomorphic function on Gc if and only i/A = /i + p£ 
p + A + . If it extends we denote the analytic extension also by ip\ and the 
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restriction of this extension to U is a spherical function tp on U. There is a shift 
in the index and we have the equality 

<P»+ P \u [ieA + . 

Conversely: Every spherical function ip^ on U (// € A + ) extends to a holomor- 
phic function ip^ on Uc and its restriction to G is a spherical function if on G. 
Then ip^G = <Pn+ P - 

Proof. Corollary 5.2.3 in [5] and Lemma 2.5 in [3J. □ 



4 The product formula for Jacobi polynomials 

We consider the compact Grassmanians M = U/K where U — SO(p + q), 
SU{p+q) or Sp(p+q) and K = SO(p) x SO(q), S(U(p) x U{q)) or Sp(p) x Sp(q), 
respectively. We write U — SU(p + q, F) with F e {M, C, H} and consider K as a 
subgroup of U(p, ¥)xU (q, F). We exclude the case p = q and assume p > q > 1. 

4-1 Remark. Note that SU(p + q,¥) is simply connected in the case F S {C,H}, 
but the group SO(p + q) is not. It has the spin group Spin(p + q) as a double 
cover. In contrast to the noncompact case some complications can occur in the 
study of compact symmetric spaces. See e.g. Proposition 1.2, Ch. VII in [?]■ 
Therefore in the study of the real case some care has to be taken. Nevertheless 
we are able to show (with some technical effort) that the same results hold in 
the real case. 

Now we apply the general theory of Section [5J to the compact symmetric 
spaces under consideration in this article. We may choose for the maximal 
abelian subspace b C u the set of all matrices Hi X € M p+q {¥) of the form 







pxp 



ix 0, 



0( p -<j) X(J 



qx(p-q) 



o 



qXq 



where x := diag(a:i, . . . , x q ) is the q x q diagonal matrix corresponding to x = 
(xi, . . .x q ) £ M. q . In the literature often a slightly different identification of b 
without the factor i is used. See eg. [7], p. 452. Our choice is dual to o = {H x : 
x € M. q }. Compare the treatise of the noncompact case in [T8]. Naturally, this 



doesn't change any result. We identify a (and its dual space a*) with K 9 via 
H x i— > x. The corresponding root system E is of type BC qi with the convention 
that zero is allowed as a multiplicity. In this way the case B q which occurs 
for F = M is included. The roots a with their multiplicities m a are given in 
the following table. The multiplicities depend on p, q and the real dimension 
d= 1,2,4 of F = R,C,H. 



root a 


multiplicity m a 


±6j 1 < i < q 


d{p - q) 


±2e t , 1 < i < q 


d-1 


±ej ± ej, 1 < i < j < q 


d 
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We will use the notation m = (7711,7712,7713) where m, (i — 1,2,3) denotes 
the multiplicity on ±ej, ±2ei or ±e< ± e^, respectively. The canonical choice of 
positive roots is E + = {ej, 2e,* : 1 < i < q} U {e, ± : 1 < t < j < q}. The Weyl 
group W , which is generated by the reflections in the hyperplanes 

H a = {x £ R q : (a,x) = 0}, 

is the hyperoctahedral group in all cases. The Weyl chamber associated to E + 
is given by 

a + = {H x : x = (xi, . . . , x q ) £ W 1 such that x% > X2 > ■ ■ ■ > x q > 0}. 
The set of dominant weights is given by 



A+={AeK« :A a G Z+ (Va e S+)} = 



(Z+)« ifF = R; 
2(Z+)« ifF = C,l 



In the case of a compact symmetric space, the affine Weyl group contains more 
information about the space. For k € 7rZ and a € E let 

k 

H a , k : (a, x) = k} = H a + -a v 

and denote by s a ,k the reflection in the hyperplane H Ut k- The affine Weyl group 
Wag is the (infinite) group generated by the reflections {s a ^ '■ a £ E, fc G 7rZ}. 
It is a semidirect product of the coroot lattice and the Weyl group W a s = 
Q v x W. 

The connected components of R 9 \ {IJ Q eE+ fce-n-z H<*,k} are called alcoves. In 
the case of BC q we choose the fundamental alcove 

A Q = {x G R g : - > £1 > x 2 > . . . > x q > 0}. (4.1) 

(The Weyl chamber a + is cut off by Q v .) Its closure Aq is a fundamental domain 
for the action of Waff on o. 

The following theorem gives a sharpened i^Aif-decomposition of U . 

4.2 Theorem. Let U = SU(p + q,¥). The group U decomposes as U = KSK, 
where 

cosx Oqx(p-q) isinx 

b x = j 0(p-<j)xg Ip— q 0(p-(j)x<} j ■ x E Aq 
isinx Ogx(p-g) cosx 

Every u £ U can be written as u — kb x k! with k,k' £ K and a unique b x £ S . 

Proof. In the cases F = C,H the group U is simply connected and the result 
follows from Theorem 8.6. in Chapter VII of [7j: Put Qo := {Hi X : x £ Aq}. 
Then a short calculation shows that S — expQo- 

In the case of SO(p+q) we cannot apply this theorem but the decomposition 
is also valid and unique. See [20], Section 15.1.9. □ 
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Now we turn our attention to the spherical functions on U . Our first aim is 
to make the product formula 

tl>(g)il>{h) = f ^{gkh)dk (4.2) 

J K 

explicit. For this, we may closely follow the argumentation of 18 (Section 2) 
in the non-compact dual cases. Since spherical functions on U = KSK are 
X-biinvariant they are determined by their values on S. We consider 

o JH° v) eKSK - 

and write g in p x q block notation as 

n _(A{g) B(g) 
9 \C(g) D{g) 

A short calculation then gives 

D(g) = vcosxv (4.3) 

where cosx = diag(cosxi, . . . ,cosa; g ) with cosxi £ [0, 1] for all i. We denote 
by spec s (X) the singular spectrum of X G M q (¥), that is 

spec s (Jf) = ^spec(X*X) = (<r 1 , ...,a q )e R q , 

with the singular values <Ji of X ordered by size: o\ > . . . > a q > 0. Equation 
(|4. 3[) implies that the singular spectrum of D(g) is given by spec s (D(g)) = 
(cos^i, . . . ,cosx q ) =: cosx. By our choice of the fundamental alcove Aq, we 
therefore have 

x = arccos(spec s (£>(.g)) Vg £ Kb x K, x £A^, (4.4) 

where arccos is also taken componentwise. 

In order to evaluate formula (|4.2[) explicitly, we write b x G S in p x q block 
notation: 

b x =( Ax Bx 

Then for k = ^ G K we obtain by a short calculation that 

D(b x kb y ) — — sin x <Jq u<jq sin y + cosxweosy. 
with the p x q block matrix 



a := 



9 



Now let ijj be a spherical function on U and put ij){x) '■= ^(bx) for x E A . 
From (|4.4p it follows that i/j satisfies 

ij)(x)ij)(y) = / V (arccos (spec s (D(b x kb y ))) dk. (4.5) 
Jk 

For our later extension of this product formula beyond the geometric cases, it 
is important to rewrite it in a way where the parameter p is no longer contained 
in the domain of integration. Under the technical assumption p > 2q, this can 
be done in the same way as in [TH], which leads to the following 

4.3 Proposition. Suppose that p > 2q. Define 

B q := {w G M q (¥) : w*w < I}, 

and for ji € C with Re \i > 7 — \, put 

Kfl := / A(7 - w*w)^ 1 dw 

where A is the usual determinant on M q (W) for F = R or C, and the Dieudonne 
determinant for ¥ = H. i.e. A(X) = (detc(X)) 1 / 2 , when X is considered as a 
complex matrix. 

Then the spherical functions ^p{x) — ip(b x ) (x € Aq) on U satisfy the product 
formula 

ip(x)ip(y) — — - — / I ip (arccos (spec s (— sin xws'my + cos 2; v cosy))) 

Kpd/2 JB q Ju (q,¥) ~ 

■ A(7 - w*w) pd/2 ~~<dvdw. 

The next step is to identify the spherical functions on SU(p + q,¥) as 
Heckman-Opdam Jacobi polynomials of type BC q . 

Consider first the general situation. Let G be a simply connected, semisimple 
noncompact Lie group with maximal compact subgroup K. Let q, 6 be the Lie 
algebras of G and K with corresponding Cartan decomposition g = 6 © p. 
Choose a maximal abelian subspace a C p and denote by E = 0) the 
(restricted) root system with geometric multiplicity m. Then the spherical 
functions on G are indexed by Oc, ([!]), where two spherical functions ip\ and 
(ffj, coincide iff A and [i are in the same orbit of the Weyl group W(Yj). The 
following important fact links the theory of Heckman and Opdam with the 
classical theory of symmetric spaces: 

4.4 Proposition, (f^, Theorem 5.2.2) Let (p\, A G etc, be a spherical func- 
tion on G, let S and m be the associated restricted root system and geometric 
multiplicity. Then for x G o, 

ipx(expx) = F\{m;x). 
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We know from Proposition 13.21 that a spherical function ip^ on G extends 
holomorphically to the complexification Gc if and only if ji € p + A + . In this 
case the restriction to the simply connected U is a spherical function ipx on [/, 
where 

i>\ = tp\+ P \u, A G A+. 
When combining this with Proposition 14.41 and (|2.3p . we obtain 

4.5 Theorem. Let U/K be a compact symmetric space with dual G j K . Assume 
that U is simply connected and let (a, S, m) be the associated geometric root data. 
Then the spherical functions onU - restricted to iexpa - are (Heckman-Opdam) 
Jacobi polynomials of type £ and with multiplicity m: 

ift\(iexpx) = ip\+ p (iexpx) = F\ +p (m;ix) = R\(m;x), for all A <G A + , x € a. 

Here we denote by ip the spherical functions on U and by ip the holomorphic 
extension of the spherical functions on G to the complexification Gc ■ 

The second equality in the theorem above follows from Proposition ^. 41 since 
px+p and F\ +p extend holomorphically to the complexification. 

4.6 The real case U = SO(p + q). As already remarked, the case U = 
SO(p + q) has to be treated with some care, since this group is not sim- 
ply connected. If U is not simply connected then we only know that the set 
A + (U/K) of highest weights of if-spherical representations of U is a sublattice 
of {v € a : v a e Z+ _(Va e £+)}. 

The spin group U = Spin(p + q) is the universal covering group of U = 
SO(p + q). The spin group is a subset of the Clifford algebra. Denote by 
ip : U — » U the covering homomorphism. The kernel S :— ker tp = {±1} is con- 
tained in the center of U. Further, there exists a unique involution 9 : U — > U 
which corresponds to the involution 6 : u — > u on the Lie algebra. Since U 
is simply connected, the group K := U 9 of fixed points of 9 is connected ([7], 
Theorem 8.2 in Chapter VII). 

We claim that 

A + (U/K) = A+(U/K) = {i/ea:i/ a eZ + (Vae £+)} . 

This is equivalent to showing that every if-spherical representation on U de- 
scends to a AT-spherical representation on U. Note that in general if a K- 
spherical representation descends it will have automatically a /To-fixed vector, 
but not necessarily a if -fixed vector (See the remarks in [M] after Theorem 
2.1.). Since K = SO(p) x SO(q) is connected this problem doesn't occur in our 
case. The fact that every if-spherical representation on U descends will follow 
as soon as we know that S C K. For this, recall that (U, K) is a Riemannian 
symmetric pair and that 9 : U — > U is an involution of U which corresponds 
to the Cartan involution on the Lie algebra u. Thus Lemma 1.3, Ch. VII, [7] 
implies 9(S) C S. Since 9 is an involution we have 9(1) — 1 and 9(—l) = — 1. 
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Therefore S is contained in the fixpoint group K. 

From this the claim follows either by a remark of Vretare (The spherical func- 
tion on U coincides with those on U for which tp(u's) = ip{u'), vl € U, s € S; see 
[19) . p. 357.) or by the following reasoning: A spherical representation n on U is 
irreducible. Since S is contained in the center of U it acts by a constant (Schur's 
Lemma). This constant has to be 1 because every i^-fixed vector is also S- fixed. 

We summarize: The spherical functions on U coincide with those on U. 

Using Theorem 14.51 and 14.61 we conclude for the spherical functions ip\ (A € 
A + ) on our compact Grassmann manifolds U/K over each of the (skew-) fields 
Fe{R,C,H}: 

$\(x) = ipxQ>x) = Fbc„ (A + p, m; ix) = R\(x). (4.6) 
Under the assumption p > 2q we get from Proposition 14.31 

Rx(x)R x (y) = — !— • / / R\ (d(xy, v, w)) A(I ~ w*w) pd/2 -~ r dvdw, 

K pd/2 JB q JUo(q,¥) 

for all x, y G M 9 , where 

d(x,y,v,w) := arccos (spec s (— sinx, ws'my + cos cost/)) . (4.7) 

The next step is analytic continuation. Fix q and d — dim^F. For \i G C 
with Re p, > 7 — 1 and dominant weight A define 

R%(x) := F BCq (A + p, m M ; ix), 

where the multiplicity is given by 

wij, = (2/i — dq, d — l,d). 

For p = pd/2, with p E N we have geometric cases. This means that for these 
values of p, the Jacobi polynomials i?^ are the spherical functions of a compact 
symmetric space U/K. 

4.7 Theorem. For p £ C with Re/i > 7 — 1 as above the Jacobi polynomials 
i?^ satisfy the product formula 

R%(x)R%{y) = — [ [ R x t (d(x,y,v,w))A(I-w*w)i*-' y dvdw 

K M JB q JUo(q,¥) 

with g?(- • • ) as in (14.71) . 

Proof. The proof is a direct copy of the first part of the proof of Theorem 4.1 
in [18] . Replace the (R, Annotation by our (S, m)-notation and the product 
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formula by our product formula. Then rewrite the claimed formula in terms of 

OH ._ 1 
A ' c(A+p,m M ) A 



P\ := c(A+ 1 m ) (^he standard Heckman-Opdam normalization): 



Pg(x)P£{y) = ^ r / / P£ (d(x, y, v, w)) A{I-w* w y-~< dvdw 



For fixed A e A + , the function c(X+p, m^) is bounded away from zero as p — > oo 
in the half plane JJ = j/i £ C : Re > 7 — 1} (see [18]). Then one uses the 
fact that the coefficients of the P£ with respect to the monomial basis {e u : 
v 6 A} are rational (see [13] , Par. 11), and that the integral 

— / lA^-w*™)^ 7 !^ 
K n Jb, 

converges exactly for Re fi > 7 — 1 and is of polynomial growth as /1 — > 00 in 
This allows to apply Carlson's theorem. For details we refer to [IS]. □ 



5 The rank one case 

At this point it is worthwile to spend some time to see how our product for- 
mula for Heckman-Opdam Jacobi polynomials generalizes the product formula 
of classical one-variable Jacobi polynomials for certain indices. 

The classical Jacobi polynomials with indices a, /3 > —1 are given by 

p(a,f» ix) = i>±ik 2F ( a + p + n + ! a + 1; iz£) (5.1) 
n! 2 ' 

where 2-F1 is the Gaussian hypergeometric function and (a)& := a(a + 1) . . . (a-l- 
fc — 1) = F w + f • We renormalize: 

(a + l) n 

Let us consider the Heckman-Opdam theory in the rank one case. The root 
system is BC\ = {±ei,±2ei} in o = 1 and we denote the multiplicity by 
mi := m a and m 2 '■— to 2q . Note that zero is allowed as a multiplicity and in 
fact, we have m 2 = in the real case. 

According to the example in [15] . p. 89f, the hypergeometric function Fbc x 
is given by 

F BCl (A, m;x) = 2 Fi (a, b, c; - (1 - cosh 2a;)) . (5.2) 

Here 

a = -(A+-mi+m 2 ), 6 = -(-A+-mi+m 2 ) and c = -(l+mi+m 2 ). (5.3) 
The dominants weights in the rank one case are 

A+(E) = 
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According to Lemma |2~51 we have 

F x+p (ix) = R\(x). 

In the case F <E {C, H} let fj, = A + p = 2n + \m\ + 777,2 G p + 2Z+ and choose 
a,b,c as in (|5.3p . Then we get from equation (|5.2|) 



i^BCi (/•*, s) = 2^1 («+ + m 2 ,-n, ^(1 + mi + m 2 ); ~(1 - cosh 2a;)) 
and with equation (|5.ip we conclude 

E A (a;) = i?i Q '' 3) (cos2x) 

where 

In the real case F = R the rank one symmetric space SO(p+ \)j SO(p) is the 
sphere § p , the root system £ = {±ei} is of type -Bi and the dominant weights 
are the natural numbers. It is well known that the spherical functions of S p are 
Gegenbauer polynomials. In fact, we have in the real rank one case for a weight 
A = n € Z+ 

(77- ! 77 1 I 

2 + 2 mi ' _ 2 ' 2^ + TOl ^' 2*- 1 _ cos2x ) 

Using the hypergeometric identity (3.1.3) in pQ, we obtain 

/ 1 . 1 — cos x 
R\(x) = 2-F1 I n + mi, —n, -(1 + mi); 

= i#*' a >(cosz), 

where a = h[mi — 1). So the Heckman-Opdam Jacobi polynomials on the 
sphere are indeed Gegenbauer polynomials. 

To summarize: The spherical functions of the compact rank one symmetric 
space U/K are given by (classical) Jacobi polynomials. This is well known, see 
e.g. Theorem 4.5, Ch. V in [8]. 

In the 1970ies, Koornwinder devoted a series of papers to the product for- 
mula for one- variable Jacobi polynomials, see e.g. |12j . For arbitrary a > (3 > 
— i, it is given by 

R^\x)R^\y) = f Q f\^\\{l + x)(l + y) + \{l -x)(l- v y 



+ yl — x 2 \/l — y 2 r cos 0—1) dm a ^(r, 6) 

with 

dm a , {r,6) = c Qi/3 (l - r 2 ) ^" 1 ^ sin 6) 2p rdrd9 
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and 

— = / / (l-r 2 ) Q ~ /3 ~ 1 (r sin 6) 2l3 rdrd6. 

Ca,P Jo JQ 

Now consider the product formula from Theorem l4.7l in the geometric cases 
for rank q = 1. The geometric multiplicities are 

mi = d(p — 1) and TO2 = d — 1, 

where e? = dirngF e {1,2, 4} and p g N, p > 2. Then we use the identification 
[[5"]) with parameters 

a = — (mi + m2 ~ 1) = —dp — 1 anc ^ ^ = 2^ m2 ~ = 2^ _ ^ 

The domains of integration reduce to Bi = {w e F : |w| < 1} and Uq(1) = 
{v £ F : |w| = l}o- Furthermore we have 

d(x, y,v,w) = arccos | — w sin x sin y + v cos x cos y\ 

and A(/-w;*u;)^ = (1 - {w] 2 )^-^- 1 . The [7 (l)-integral cancels under the 
coordinate transform w' := v~ 1 w. Using cos2x = 2 cos 2 x-lwe obtain 

R^ ) (cos2x)R^\cos2y) = 

— / i?i Q ' /3) (2|-zsina;siny + cosa;cosy| 2 -l)-(l-|z| 2 )^ p - 1) - 1 rfz. 

K pd/2 J Bi 



(5.4) 



Let us sketch the further calculations only in the case F = C. We introduce 
polar coordinates z = re 19 and put t := cos2x, s := cos2y. Then use the 
identities sin 2 x = i(l — i), sinxcosx = h\/l — t 2 and cos 2 x — |(1 + t). The 
constant K p[ j/2 1S given by 

Kpd/2 = 2ir [ {l-r 2 y- 2 rdr 



p-1 

We conclude from (|5.4p exactly the product formula for Jacobi polynomials with 
a = p — 1 and j3 = 0: 



o Jo 



+ y/l-t 2 ^l-s 2 rcos6~ 1)(1 - r 2 ) p - 2 rdrd6. 



6 Hypergroup structures on the alcove 

In this section we shall see that the product formula of Theorem 14.71 leads to 
three continuous series (for d = 1, 2, 4) of positivity-preserving convolution alge- 
bras on the fundamental alcove Aq = {t € E 9 : « > ti > . . . > t q > 0}, which 
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are compact commutative hypergroups with normalized Jacobi polynomials as 
characters. In the geometric cases {p, — pd/2), these hypergroups convolutions 
are just given by the double coset convolutions on the double coset space U//K 
which may be identified with Aq by Theorem 14.21 

To start with, let us briefly recall some basics from hypergroup theory. For 
a detailed treatment, the reader is referred to [11) . Hypergroups generalize the 
convolution algebras of locally compact groups, with the convolution product 
of two point measures S x and d y being in general not a point measure again but 
a probability measure with compact support depending on x and y. 

6.1 Definition. A hypergroup is a locally compact Hausdorff space X with 
a weakly continuous, associative convolution * on the space Mb(X) of regular 
bounded Borel measures on X, satisfying the following properties: 

(i) The convolution product S x * S y of two point measures is a compactly 
supported probability measure on X, and supp(<5 x * 8 y ) depends contin- 
uously on x and y with respect to the so-called Michael topology on the 
space of compact subsets of X (see [TTj). 

(ii) There exists a (necessarily unique) neutral element e G X satisfying 
S e * S x — 5 X * S e — 6 X for all x G X. 

(iii) There exists a (necessarily unique) continuous involution x <— > x on X 
such that 8 X * 5 y = {5 y * 5 X )~ and x = y e G supp((5 x * 8 y ). (Here 
the measure /i - is given by pT (A) = p(A).) 

The hypergroup is called commutative if the convolution is commutative. 

Note that due to weak continuity, the convolution of measures on a hyper- 
group is uniquely determined by the convolution of point measures. 

Every commutative hypergroup X has a unique (up to a multiplicative fac- 
tor) Haar measure w, that is a positive Radon measure with the property 

/ f(x*y)Mv)= I f(v)Mv) (Vxexj eC c (x)), 

where we use the notation f(x * y) := (5 X * S y )(f). 
The dual space of a hypergroup X is defined by 

X := {(p G C b (X) : ip ^ 0, (p(x) = tp{x) and ip(x *y) = ip(x)(p(y)}. 

The elements of X are called characters of X. As in the case of locally compact 
abelian groups, the dual of a commutative hypergroup is a locally compact 
Hausdorff space with the topology of locally uniform convergence. In general 
the dual is not again a hypergroup. In the case of a compact hypergroup X the 
dual X is discrete. The Fourier transform on L X (X, u>) is defined by f(<p) := 
Jx f{x)^p{x)du){x). It is injective and there exists a unique positive Radon 
measure i on X, called the Plancherel measure of (A,*), such that />—»•/ 
extends to an isometric isomorphism from L 2 (A, oj) onto L 2 (X,ir). 
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6.2 Example. (Double coset hypergroups) 

Let G be a locally compact group with compact subgroup K and denote by dk 
the Haar measure on K. Then there is a natural hypergroup structure on the 
set of double cosets G//K = {KxK : x £ G} which is given by 



KxK * S Ky K = / 5 KxkyK dk, x, y e G. 

J K 



The neutral element is K — KeK and the involution is given by (KxK)~ = 
Kx~ Y K (see Theorem 8.2B in [TT]). The double coset hypergroup (G/ / K, *) is 
commutative if and only if (G,K) is a Gelfand pair. 

6.3 Theorem. Let /16C with Re (j, > 7 — 1. Then the probability measures 
— / I f (arccos (spec s (— sinxw sin y + cos xv cosy))) • 

K p JB q Juo(q.F) ~ ~ 

■ A(I - w*wY~ 1 dvdw 

for x,y G Aq define a commutative hypergroup structure on the compact alcove 
Aq. The neutral element is and the involution is the identity mapping. 

Note that in the geometric cases /1 = pd/2 the hypergroup on Ag coincides 
with the double coset hypergroup U//K. 

Proof. Commutativity is obvious. For associativity let x,y, z G Aq. Then 
8x*n(8y*n8z){f) =~2 I I f(D{x,y,z,v,w,v',w'))- 

K fj, JB q xU (q) JBgXUo(q) 

■ A(I - w*w) t *~' ( A(I - {w')*w'Y~ 1 dvdwdv'dw l =: 

with a certain Aq valued argument D, which is independent of /1. The same is 
true for 

(S x * M S y ) * M S z (f) =: I'(fi) 

with a /^-independent argument D 1 instead of D. The integrals and /' (/i) 
are well defined and holomorphic in {// G C : Re fi > 7 — 1}. The convolu- 
tion is associative in the geometric cases [i = pd/2. Analytic continuation yields 
associativity for all \i with Re fi > 7 — 1 as in [17] . Weak continuity of the convo- 
lution follows from the continuity of the mapping (x, y, v,w) <— ► f(d(x,y,v,w)) 
on Aq x B q x Uo(q,¥). The compact support of S x * S y is trivial. It is also 
obvious that is neutral. So only the support continuity with respect to the 
Michael topology and the fact that the identity mapping is a hypergroup in- 
volution remain. As the support of 5 X * M 5 y is independent of /1, it suffices to 
verify both statements in the geometric cases U//K. But these are known to 
correspond to double coset hypergroups, which immediately implies the support 
continuity. In the geometric cases, the involution is induced from the mapping 



17 



x i— > —a; on R 9 = a. A short calculation shows that 6_ a € Kb x K on U / /K and 
therefore the involution on U / /K is the identity. In fact, 

cos x z sin x ' 

, / sin ,r COS X 

— I q \ / cosx — isinx 

l-n-la.a In-n Ij 





p-2q,q I | J p-q I I 1 p-2q,q 

-isinx cosx / V I q . 

where l n ,m — diag(l, . . . , 1, —1, . . . , —1) denotes the diagonal matrix with the 
first n entries equal to 1 and the last m entries equal to —1. Then 

det (~ Iq T ) = 1. □ 

V 1 p-2q,qJ 

6.4 Proposition. The support of 5 X * M 5 y satisfies 

supp((5 K * M S y ) C {z G A Q : IjzHoo < Hxlloo + ||y||oo} 
where \\ ■ is the maximum norm in M. q . 
Proof. For a matrix A £ M q (¥) we denote by 



v / spec(A*A) = spec, (A) = {<t,(A) : i = l,...,g} 

the singular values of A, decreasingly ordered by size: ci(A) > . . . > o- q (A) > 0. 
Write ||A|| = ||spec s (A)|| 00 = o"i(A) for the spectral norm of A. Recall the 
following estimates from Theorem 3.3.16 in [9]: 

\a q (A + B)-a q (A)\<a 1 (B) (6.1) 
<? q (AB) <a q (A)a 1 (B) (6.2) 

These estimates are only given for F = K, C but in the case of a quaternionic 
matrix A € Af g (H) we simply consider the corresponding complex matrix \a G 
M 2(? (C), namely 

Ai A 2 N 



XA = ■ -A 2 A x 

where Ai, A 2 are complex g x g-matrices such that A = A\ + A 2 j. The map 
A/ g (H) — > A/ 2(? (C), A i—* xa is a homomorphism and xa* — (xa)* ■ Moreover, 
spec s (A) = spec s (xA) where in the second set each singular value appears twice 
(see [21] for a survey about quaternionic matrices). 
Let £ := cosx v cosy — sinx w siny. By (16.11) . 



fq(£) > 0\j(cosxi> cosy) — cri(sinxw siny). 
Since sinx is increasing on [0, 7r/2] we get (using submultiplicativity) 

ai (sinx w siny) = || sinx w siny|| < || sinx|j|| sin y | — sin ||x||oo sin ||y||oo- 
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On the other hand, if yi ^ for all i, then by 

. . a„(coaxv) 

aJcosxv cosy) > — f- — T > cos ||x||oo cos ||y||oo- 

cti ((cosy) l ) 

Therefore 

<7q(Q > cos Hxlloocos \\y\\oo - sin H^Hoo sin \\y\\oo = cosfllxHa, + \\y\\oo) 

This implies the claim, because arccos is decreasing. If yi — for some i, 
the estimate remains by continuity since the eigenvalues of a matrix depend 
continuously upon its entries; see e.g. [9], p. 396.) 

□ 

Because of Theorem 14.71 the Jacobi polynomials R\ are multiplicative, 
R\(x)R\(y) = R\(x * M y). 

6.5 Lemma. Assume that the Weyl group W contains the reflection a : x i— > 
— x. Then the Jacobi polynomials R\ are real-valued on R 9 . In particular, this 
holds for the root systems B q , C q and BC q . 

Proof. The polynomial R\ is a linear combination of the orbit sums M^, where 

Since a S W we sum up e* 71 + e~ 111 + . . .. This is real- valued. □ 

Therefore the R\, A e A+ are characters of the hypergroup (A , *). In fact 
they are all: 

6.6 Proposition, (a) The Haar measure of the hypergroup (Aq,*^) is given 
by 



dcj(x) = w m (x)dx = J] \e l{a - x) - e~ l{a - x) 



dx. 



(b) The dual space is (A ,* M ) A = {R\ : A E A+}. 

Proof, (a) For a Jacobi polynomial R\ with A^Owe have j-^R\duo — since 
R\ is orthogonal to Rq = 1 with respect to (-,-) m . For z e Aq consider the 
generalized translation 

(r z f)(x) := f(z * M x) = _f(y)d(S z * M 5 x )(y). 

In view of ([6|), we obtain 

[_(T z R x )(x)duj{x) = j_Rx{z x)du(x) = Rx{z) f_R x {x)dw{x) = 0. 

J Aq J Aq J A 
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By linearity, the above equation holds for all VF-invariant trigonometric poly- 
nomials. Note that T w is dense in C(Ao) with respect to the norm || • 
(Stone- Weierstrass). Now the assertion follows from the || • Hoo-continuity of the 
generalized translation (see Lemma 3.3B in [TT]). 

(b) We already know that the R\ are characters of our hypergroup. In 
general, the characters of a compact hypergroup X form an orthogonal basis of 
L 2 (X,duj) (The proof is the same as in the case of a compact group and uses 
the Plancherel Theorem. See Theorem 3.5 in [S]). The Jacobi polynomials form 
already an orthogonal basis of L 2 (Aq). So there are no additional characters. □ 

6. 7 Remark. For a general commutative hypergroup X the set of bounded semi- 
characters 

Xb{X) := {ip S C b (X) : <p ^ and ip(x *y) = (p(x)<p(y)}. 

may not coincide with the dual X . However, in the case of a compact commu- 
tative hypergroup X (or more general a commutative hypergroup of subexpo- 
nential growth), one has X — Xb(X); see Theorem 2.5.12 in [2J. But Lemma 
16.51 is much simpler and of some interest on its own. 

We identify the dual of our hypergroup with the set of dominant weights via 
the mapping (^o) A - * A + , R\ i— ► A. 

6.8 Proposition. The Plancherel measure of the hypergroup (Aq,*^) is the 
following measure on A + ; 

where <5„ denotes the point measure in v G A + . 

Proof. The set {r]/ 2 ^ : v 6 A + } is an orthonormal basis of L 2 (Aq,(jj). Thus 
for / G L 2 (Aq,oj) we obtain 




□ 
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